Systems and methods for integrating imagery captured by different imaging modalities into composite imagery of a surgical space

ABSTRACT

An exemplary system accesses imagery of a surgical space captured by different imaging modalities and, based on the accessed imagery, generates composite imagery that includes integrated representations of the surgical space as captured by the different imaging modalities. An exemplary composite image includes a representation of the surgical space as captured by a first imaging modality augmented with an integrated representation of the surgical space as captured by a second imaging modality. The integrated representation of the surgical space as captured by the second imaging modality may be selectively movable and may be generated based on first imagery of the surgical space captured by the first imaging modality and second imagery of the surgical space captured by the second imaging modality in a manner that provides a visually realistic appearance of depth.

RELATED APPLICATIONS

The present application claims priority to U.S. Provisional PatentApplication No. 62/855,755, filed on May 31, 2019, and entitled “SYSTEMSAND METHODS FOR INTEGRATING IMAGERY CAPTURED BY DIFFERENT IMAGINGMODALITIES INTO COMPOSITE IMAGERY OF A SURGICAL SPACE,” the contents ofwhich are hereby incorporated by reference in their entirety.

BACKGROUND INFORMATION

During a surgical procedure, an endoscope may be used to captureendoscopic imagery of a surgical space. The endoscopic imagery may bepresented to a surgeon by way of a display device so that the surgeonmay visualize the surgical space while performing the surgicalprocedure. An endoscope is one imaging modality that is used to captureimagery of the surgical space.

In some scenarios, one or more additional imaging modalities may be usedto capture additional imagery of the surgical space that may also bepresented to the surgeon. For example, an ultrasound scan, acomputerized tomography (CT) scan, and a magnetic resonance imaging(MRI) scan are other imaging modalities that may be used to captureimagery of the surgical space.

Imagery captured by different imaging modalities may be presented to thesurgeon such that the surgeon may visualize the surgical space whileperforming the surgical procedure. However, there remains room toimprove technologies for processing and presenting imagery captured bydifferent surgical space imaging modalities.

SUMMARY

The following description presents a simplified summary of one or moreaspects of the systems and methods described herein. This summary is notan extensive overview of all contemplated aspects and is intended toneither identify key or critical elements of all aspects nor delineatethe scope of any or all aspects. Its sole purpose is to present one ormore aspects of the systems and methods described herein as a prelude tothe detailed description that is presented below.

An exemplary system includes a memory storing instructions and aprocessor communicatively coupled to the memory and configured toexecute the instructions to determine an image render viewpoint fromwhich to render an image of a surgical space; determine, from aperspective of the image render viewpoint, a position of an augmentationregion relative to the surgical space, the augmentation regionselectively movable relative to the surgical space; generate a compositeimage of the surgical space from the perspective of the image renderviewpoint and based on the determined position of the augmentationregion relative to the surgical space, and direct a display device todisplay the composite image. The composite image may include: theaugmentation region at the determined position of the augmentationregion relative to the surgical space; outside the augmentation region,a representation of a first portion of the surgical space as captured bya first imaging modality; and inside the augmentation region, arepresentation of a second portion of the surgical space as captured bya second imaging modality, the representation of the second portion ofthe surgical space generated based on first imagery of the surgicalspace captured by the first imaging modality and second imagery of thesurgical space captured by the second imaging modality.

Another exemplary system includes a memory storing instructions and aprocessor communicatively coupled to the memory and configured toexecute the instructions to access first imagery of a surgical spacecaptured by a first imaging modality; access second imagery of thesurgical space captured by a second imaging modality, the second imagingmodality different from the first imaging modality; generate a compositeimage of the surgical space based on the first imagery captured by thefirst imaging modality and the second imagery captured by the secondimaging modality; and direct a display device to display the compositeimage. The composite image may include: a representation of a firstportion of the surgical space as captured by the first imaging modality,the representation of the first portion of the surgical space generatedbased on the first imagery captured by the first imaging modality; anaugmentation region integrated within the representation of the firstportion of the surgical space; and, inside the augmentation region, arepresentation of a second portion of the surgical space as captured bythe second imaging modality, the representation of the second portion ofthe surgical space including a composition of imagery of the surgicalspace captured by the second imaging modality modified by a feature ofimagery of the surgical space captured by the first imaging modality.

An exemplary method includes a computing system determining an imagerender viewpoint from which to render an image of a surgical space;determining, from a perspective of the image render viewpoint, aposition of an augmentation region relative to the surgical space, theaugmentation region selectively movable relative to the surgical space;generating a composite image of the surgical space from the perspectiveof the image render viewpoint and based on the determined position ofthe augmentation region relative to the surgical space; and directing adisplay device to display the composite image. The composite image mayinclude: the augmentation region at the determined position of theaugmentation region relative to the surgical space; outside theaugmentation region, a representation of a first portion of the surgicalspace as captured by a first imaging modality; and, inside theaugmentation region, a representation of a second portion of thesurgical space as captured by a second imaging modality, therepresentation of the second portion of the surgical space generatedbased on first imagery of the surgical space captured by the firstimaging modality and second imagery of the surgical space captured bythe second imaging modality.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate various embodiments and are a partof the specification. The illustrated embodiments are merely examplesand do not limit the scope of the disclosure. Throughout the drawings,identical or similar reference numbers designate identical or similarelements.

FIG. 1 illustrates an exemplary imaging modality integration systemaccording to principles described herein.

FIG. 2 illustrates the imaging modality integration system of FIG. 1configured to generate composite imagery of a surgical space based onimagery captured by different imaging modalities according to principlesdescribed herein.

FIG. 3 illustrates an exemplary composite image of a surgical spaceaccording to principles described herein.

FIG. 4 illustrates an exemplary positioning of an image render viewpointand an augmentation region relative to a surgical space according toprinciples described herein.

FIG. 5A illustrates an exemplary real workspace included in a surgicalspace according to principles described herein.

FIG. 5B illustrates an exemplary endoscopic image of the real workspaceof FIG. 5A according to principles described herein.

FIG. 5C illustrates an exemplary slope image extracted from theendoscopic image of FIG. 5B according to principles described herein.

FIG. 6A illustrates an exemplary virtual workspace according toprinciples described herein.

FIG. 6B illustrates an exemplary image of the virtual workspace of FIG.6A according to principles described herein.

FIG. 6C illustrates an exemplary mask image according to principlesdescribed herein.

FIG. 7 illustrates an exemplary composite image of a surgical spaceaccording to principles described herein.

FIG. 8 illustrates an exemplary computer-assisted surgical systemaccording to principles described herein.

FIG. 9 illustrates an exemplary method according to principles describedherein.

FIG. 10 illustrates an exemplary computing device according toprinciples described herein.

DETAILED DESCRIPTION

Systems and methods for integrating imagery captured by differentimaging modalities into composite imagery of a surgical space aredescribed herein. In certain examples, an imaging modality integrationsystem may be configured to integrate imagery captured by differentimaging modalities by generating composite imagery that includesintegrated representations of a surgical space as captured by multipledifferent imaging modalities. For example, the imaging modalityintegration system may be configured to generate a composite image thatincludes a representation of a first portion of the surgical space ascaptured by a first imaging modality and an integrated representation ofa second portion of the surgical space as captured by a second imagingmodality. The imaging modality integration system may be configured tointegrate the representation of the second portion of the surgical spacewith the representation of the first portion of the surgical space in amanner that augments imagery of the surgical space as captured by oneimaging modality (e.g., endoscopic imaging) with imagery of the surgicalspace as captured by a second imaging modality (e.g., ultrasound, CT, orMRI imaging).

In certain examples, the imaging modality integration system may beconfigured to generate the representation of the second portion of thesurgical space based on first imagery of the surgical space captured bythe first imaging modality and second imagery of the surgical spacecaptured by the second imaging modality. For example, the representationof the second portion of the surgical space may be generated to includea composition of the second imagery of the surgical space and a featureof the first imagery of the surgical space. For instance, to produce therepresentation of the second portion of the surgical space, the secondimagery of the surgical space may be combined with slope imageryrepresenting gradient information extracted from the first imagery ofthe surgical space. Such a composition may produce a visually realisticappearance (e.g., an appearance of depth that facilitates depthperception) of the representation of the second portion of the surgicalspace integrated with the representation of the first portion of thesurgical space. In certain examples, the composition is generated basedon actual, organic colors of the first imagery and the second imagery,without using artificial or non-photorealistic colors. Examples of howthe representation of the second portion of the surgical space may begenerated and integrated with the representation of the first portion ofthe surgical space are described herein.

In certain examples, the first imaging modality may include endoscopicimaging (e.g., imaging by an endoscope) that captures endoscopic imageryof surface anatomy included in a surgical space, and the second imagingmodality (e.g., ultrasound, CT, or MRI imaging) may capture imagery ofsubsurface anatomy included in the surgical space. In such examples, theimaging modality integration system may be configured to generatecomposite imagery that includes a representation of a first portion ofthe surgical space that is generated based on the endoscopic imagery ofsurface anatomy as captured by an endoscope and an integratedrepresentation of a second portion of the surgical space that isgenerated based on the imagery of the subsurface anatomy as captured bythe second imaging modality. The representation of the second portion ofthe surgical space may be generated to include a composition of theimagery of the subsurface anatomy as captured by the second imagingmodality and a feature of the endoscopic imagery of the surface anatomyas captured by the endoscope. For instance, the imagery of thesubsurface anatomy as captured by the second imaging modality may becombined with slope imagery extracted from the endoscopic imagery toproduce the representation of the second portion of the surgical space.Such a composition may produce a visually realistic appearance (e.g., anappearance of depth that facilitate depth perception) of the imagedsubsurface anatomy relative to the imaged surface anatomy when therepresentations are integrated in a composite image.

In certain examples, the second portion of the surgical space may bedynamically selected by way of user input to a computer-assistedsurgical system or automatically by the computer-assisted surgicalsystem (e.g., by performing an automatic scan). For example, the imagingmodality integration system may be configured to provide an augmentationregion (e.g., a virtual object representing an augmentation region) thatis selectively movable relative to the surgical space based on userinput to the computer-assisted surgical system or based on automaticmovement controlled by the computer-assisted surgical system (e.g., aspart of an automatic scan). At any given time during a surgicalprocedure, the imaging modality integration system may be configured todetermine a position of the augmentation region relative to the surgicalspace and to use the determined position of the augmentation regionrelative to the surgical space to determine the first and secondportions of the surgical space to be used to generate a composite imageof the surgical space. For example, the imaging modality integrationsystem may be configured to identify the first portion of the surgicalspace to be a portion of the surgical space that is outside of theaugmentation region from a perspective of an image render viewpoint, andto identify the second portion of the surgical space to be a portion ofthe surgical space that is inside the determined position of theaugmentation region from the perspective of the image render viewpoint.The imaging modality integration system may use the identified first andsecond portions of the surgical space to generate a composite image ofthe surgical space. As described herein, the composite image may includeintegrated representations of the first and second portions of thesurgical space as respectively captured by first and second imagingmodalities.

To illustrate an example, the imaging modality integration system may beconfigured to determine an image render viewpoint from which to renderan image of a surgical space, determine, from a perspective of the imagerender viewpoint, a position of an augmentation region relative to thesurgical space, and generate a composite image of the surgical spacefrom the perspective of the image render viewpoint and based on thedetermined position of the augmentation region relative to the surgicalspace such that the composite image includes: the augmentation region(e.g., a representation of the augmentation region) at the determinedposition of the augmentation region relative to the surgical space;outside the augmentation region, a representation of a first portion ofthe surgical space as captured by a first imaging modality; and, insidethe augmentation region, a representation of a second portion of thesurgical space as captured by a second imaging modality. Therepresentation of the second portion of the surgical space may begenerated in any of the ways described herein and may be based on firstimagery of the surgical space captured by the first imaging modality andsecond imagery of the surgical space captured by the second imagingmodality.

Systems and methods described herein may provide various advantages andbenefits. For example, systems and methods described herein mayintegrate imagery captured by different imaging modalities intocomposite imagery of a surgical space in a manner that produces, withinthe composite imagery, an integrated and visually realistic appearanceof imagery of the surgical scene as captured by the different imagingmodalities. Systems and methods described herein may present thegenerated composite imagery to a user of a computer-assisted surgicalsystem, such as a surgeon utilizing the computer-assisted surgicalsystem to perform a surgical procedure. The presented composite imagerymay be visually realistic and intuitive to the surgeon, may reduce thecomplexity of the surgical procedure for the surgeon (e.g., byeliminating the need for the surgeon to mentally align imagery of thesurgical space that is presented separately in a non-integrated manner),may allow the surgeon to concurrently, conveniently, and intuitivelyvisualize surface and subsurface anatomy integrated in compositeimagery, and/or may allow the surgeon to provide input to convenientlyand dynamically select a portion of a surgical space that is to beaugmented such that the selected portion may be viewed using a differentimaging modality than is used to view another portion of the surgicalspace (e.g., by selecting a portion of the surgical space at whichimagery of subsurface anatomy is displayed as an augmentation to imageryof surface anatomy being displayed). Additionally, composite imagerythat is generated based on actual, organic colors of captured imagery,without using artificial or non-photorealistic colors, may be morerealistic in appearance (e.g., facilitate better depth perception)compared to composite imagery that is generated based on artificial ornon-photorealistic colors.

These and other advantages and benefits of systems and methods describedherein will be made apparent herein.

FIG. 1 illustrates an exemplary imaging modality integration system 100(“system 100”) configured to integrate imagery captured by differentimaging modalities, including by using the captured imagery to generatecomposite imagery that includes integrated representations of portionsof a surgical space as captured by the different imaging modalities.System 100 may be included in, implemented by, or connected to one ormore components of a computer-assisted surgical system. For example,system 100 may be implemented by one or more components of acomputer-assisted surgical system. As another example, system 100 may beimplemented by a stand-alone computing system communicatively coupled toa computer-assisted surgical system. An exemplary computer-assistedsurgical system is described further below.

As shown in FIG. 1, system 100 may include a storage facility 102 and aprocessing facility 104 selectively and communicatively coupled to oneanother. Each of facilities 102 and 104 may include or be implemented byone or more physical computing devices including hardware and/orsoftware components such as processors, memories, storage drives,communication interfaces, instructions stored in memory for execution bythe processors, and so forth. Although facilities 102 and 104 are shownto be separate facilities in FIG. 1, facilities 102 and 104 may becombined into fewer facilities, such as into a single facility, ordivided into more facilities as may serve a particular implementation.In some examples, each of facilities 102 and 104 may be distributedbetween multiple devices and/or multiple locations as may serve aparticular implementation.

Storage facility 102 may maintain (e.g., store) executable data used byprocessing facility 104 to perform one or more of the operationsdescribed herein. For example, storage facility 102 may storeinstructions 106 that may be executed by processing facility 104 toperform one or more of the operations described herein. Instructions 106may be implemented by any suitable application, software, code, and/orother executable data instance. Storage facility 102 may also maintainany data received, generated, managed, used, and/or transmitted byprocessing facility 104.

Processing facility 104 may be configured to perform (e.g., executeinstructions 106 stored in storage facility 102 to perform) variousoperations associated with integrating imagery captured by differentimaging modalities into composite imagery of a surgical space. Forexample, processing facility 104 may be configured to generate compositeimagery that includes integrated representations of a surgical space ascaptured by multiple different imaging modalities. For instance,processing facility 104 may be configured to generate a composite imagethat includes a representation of a first portion of the surgical spaceas captured by a first imaging modality and a representation of a secondportion of the surgical space as captured by a second imaging modality.Processing facility 104 may be configured to integrate therepresentation of the second portion of the surgical space with therepresentation of the first portion of the surgical space such that therepresentations become an integrated whole within the composite image.Processing facility 104 may perform the integration in any of the waysdescribed herein and in a manner that augments imagery of the surgicalspace as captured by one imaging modality (e.g., endoscopic imaging)with imagery of the surgical space as captured by a second imagingmodality (e.g., ultrasound, CT, or MRI imaging).

These and other operations that may be performed by processing facility104 are described herein. In the description that follows, anyreferences to operations performed by system 100 may be understood to beperformed by processing facility 104 of system 100.

FIG. 2 illustrates a configuration 200 in which system 100 is configuredto generate composite imagery of a surgical space based on imagerycaptured by different imaging modalities. As shown, configuration 200may include multiple imaging modalities 202 (e.g., imaging modalities202-1 and 202-2) configured to capture imagery 204 (e.g., imagery 204-1captured by imaging modality 202-1 and imagery 204-2 captured by imagingmodality 202-2) of a surgical space 206.

Surgical space 206 may include any volumetric space associated with asurgical procedure. For example, surgical space 206 may include any partor parts of a body of a patient, such as anatomy 208 (e.g., tissue,etc.) of the patient in a space associated with the surgical procedure.Surgical space 206 may, in certain examples, be entirely disposed withinthe patient and may include a space within the patient near where asurgical procedure is planned to be performed, is being performed, orhas been performed. For example, for a minimally invasive surgicalprocedure being performed on tissue internal to a patient, surgicalspace 206 may include the tissue, anatomy underlying the tissue, as wellas space around the tissue where, for example, surgical instrumentsbeing used to perform the surgical procedure are located. In otherexamples, surgical space 206 may be at least partially disposed externalto the patient. For instance, for an open surgical procedure beingperformed on a patient, part of surgical space 206 (e.g., tissue beingoperated on) may be internal to the patient while another part ofsurgical space 206 (e.g., a space around the tissue where one or moresurgical instruments may be disposed) may be external to the patient.Surgical space 206 may include a real workspace in which a surgicalprocedure is performed, such as an actual, real-world workspaceassociated with a patient and in which one or more surgical instrumentsare used to perform the surgical procedure on the patient.

As used herein, a surgical procedure may include any medical procedure,including any diagnostic or treatment procedure in which manual and/orinstrumental techniques are used on a patient to investigate or treat aphysical condition of the patient. A surgical procedure may refer to anyphases of a medical procedure, such as preoperative, operative (i.e.,intraoperative), and postoperative phases of a surgical procedure.

Imaging modalities 202 may be configured and/or used to capture imagery204 of surgical space 206. Such a capture is represented by dashed lines210 in FIG. 2. Imaging modalities 202 may each capture imagery 204 ofsurgical space 206 in any suitable manner and at any suitable time.Accordingly, one or more imaging modalities 202 may capture imagery 204of surgical space 206 during one or more preoperative, intraoperative,and/or postoperative phases of a surgical procedure.

Imaging modalities 202 may include any set of different imagingmodalities that may be used to capture imagery of a surgical space.Examples of imaging modalities 204 include, without limitation,endoscopic imaging by an endoscope, ultrasound imaging by an ultrasoundmachine, CT imaging by a CT machine, and MRI imaging by an MRI machine.Any suitable additional or alternative imaging modalities may be used inother examples. In certain implementations, imaging modality 202-1 mayinclude endoscopic imaging by an endoscope, and imaging modality 202-2may include any different imaging modality such as ultrasound imaging byan ultrasound machine, CT imaging by a CT machine, or MRI imaging by anMRI machine. In such implementations, imaging modality 202-1 may captureimagery 204-1 that is endoscopic imagery of surgical space 206, andimaging modality 202-2 may capture imagery 204-2 that is ultrasoundimagery, CT imagery, or MRI imagery of surgical space 206.

In certain examples, imaging modality 202-1 may be configured to captureimagery of surface anatomy included in surgical space (e.g., an outersurface of tissue included in the surgical space), and imaging modality202-2 may be configured to capture imagery of subsurface anatomyincluded in the surgical space (e.g., subsurface tissue that is behindthe outer surface of tissue included in the surgical space). Forexample, imaging modality 202-1 may include endoscopic imaging by anendoscope that captures images of surface tissue within a patient, andimaging modality 202-1 may include ultrasound, CT, or MRI imaging thatcaptures images of subsurface tissue that, from the perspective of theendoscope, is behind and hidden from the view of the endoscope by thesurface tissue within the patient.

As mentioned, imaging modalities 202 may each capture imagery 204 ofsurgical scene 206 at any suitable time, such as during any phase(s) ofa surgical procedure. In certain examples, imaging modalities 202 mayconcurrently capture imagery 204 of surgical space 206. For instance,imaging modality 202-1 may capture endoscopic imagery during a surgicalprocedure (e.g., during an operative phase of the surgical procedure),and imaging modality 202-1 may concurrently capture another type ofimagery during the surgical procedure. In other examples, imagingmodalities 202 may capture imagery 204 of surgical space 206 atdifferent times and/or during different phases of the surgicalprocedure. For instance, imaging modality 202-1 may capture endoscopicimagery during an operative phase of the surgical procedure, and imagingmodality 202-2 may capture another type of imagery during a preoperativephase of the surgical procedure.

Imagery 204 of surgical space 206 may include images of surgical space206 captured by imaging modalities 202. For example, imagery 204 mayinclude endoscopic images, ultrasound images, CT images, MRI images,and/or any other suitable form of images of surgical space 206. Imagery204 may include any suitable type of images represented by data in anysuitable data format. For example, imagery 204 may include still-frameimages, video, color images, infrared images, and/or any other type ofimages that may visually represent surgical space 206. An image capturedby an imaging modality may include a grid of pixels having values (e.g.,color values, brightness values, etc.) representative of an appearanceof surgical space 206 as captured by the imaging modality. Color valuesfor pixels in a captured image may represent actual, organic colors ofthe surgical space as captured by an imaging modality.

Additionally or alternatively, imagery 204 may include one or moremodels of surgical space 206 that are generated based on imagingperformed by an imaging modality. For example, imagery 204 may include athree-dimensional (3D) model of surgical space 206 that is generatedbased on imaging performed by an imaging modality, such as imagingperformed by an ultrasound machine, a CT machine, an MRI machine, orother suitable imaging modality. The 3D model may be a full volumetricmodel that includes voxels (i.e., volumetric pixels) having values(e.g., color values, brightness values, etc.) representative of anappearance of surgical space 206 at 3D points within the model. Such avolumetric model may facilitate any slice of the 3D model beingidentified and used by system 100 to produce an image of the slice ofthe 3D model. Color values for pixels in the slice image may representactual, organic colors of the surgical space as captured by an imagingmodality.

While FIG. 2 depicts two imaging modalities 202-1 and 202-2 respectivelycapturing imagery 204-1 and 204-2 that are provided as input to system100, other examples may include any suitable number and/or configurationof multiple, different imaging modalities that capture imagery that isprovided as input to system 100 for use in generating composite imageryof surgical space 206. For example, three or more different imagingmodalities may capture imagery that is input to system 100 for use ingenerating composite imagery of surgical space 206.

System 100 may generate composite imagery 212 of surgical space 206based on imagery 204 captured by imaging modalities 202. System 100 maydo this in any of the ways described herein to generate a compositeimage that includes integrated representations of portions of surgicalspace 206 as captured by different imaging modalities 202. Examples ofsuch composite images and how the composite images may be generated aredescribed herein.

System 100 may direct a display device 214 to display composite imagery212. For example, system 100 may provide data representative ofcomposite imagery 212 to display device 214, which may be configured todisplay composite imagery 212 for viewing by a user of acomputer-assisted surgical system. Display device 214 may include anydevice capable of receiving and processing imagery data to display oneor more images. To this end, display device 214 may include one or moredisplay screens on which images may be displayed. In certain examples,display device 214 may be a component of or communicatively connected toa computer-assisted surgical system.

FIG. 3 illustrates an exemplary composite image 300 that may begenerated by system 100 and displayed by a display device. Compositeimage 300 may be an image of a surgical space and may include arepresentation 302 of an augmentation region positioned relative to thesurgical space. The positioning of augmentation region 302 withincomposite image 300 may represent and/or be determined from a positionof augmentation region 302 (e.g., a position of a virtual objectrepresenting augmentation region 302) relative to the surgical space.Examples of positioning of augmentation region 302 relative to thesurgical space are described herein.

Augmentation region 302 may be any suitable shape that defines an areawithin composite image 300. For example, augmentation region 302 may bea circle, an oval, a quadrilateral (e.g., a rectangle, a fan), atriangle, or any other suitable shape.

The positioning of augmentation region 302 within composite image 300may define two portions of composite image 300—a first portion that isoutside augmentation region 302, and a second portion that is insideaugmentation region 302. In the first portion of composite image 300,which is outside augmentation region 302, composite image 300 mayinclude a representation 304 of the first portion of the surgical spaceas captured by a first imaging modality. For example, representation 304may include imagery of the surgical space captured by the first imagingmodality, such as imagery 204-1 captured by first imaging modality202-1. In the second portion of composite image 300, which is insideaugmentation region 302, composite image 300 may include arepresentation 306 of the second portion of the surgical space ascaptured by a second imaging modality. For example, representation 306may include imagery of the surgical space captured by the second imagingmodality and modified by a feature of imagery of the surgical spacecaptured by the first imaging modality, such as imagery 204-2 capturedby second imaging modality 202-2 and modified by a feature of imagery204-1 captured by first imaging modality 202-1. Accordingly,representation 306 of the second portion of the surgical space may begenerated based on both imagery 204-1 and imagery 204-2 and may includea composition of imagery 204-2 and a feature (e.g., gradientinformation) of imagery 204-1. As indicated herein, this composition maycreate a visually realistic appearance of depth of representation 306when representation 306 is integrated with representation 304 incomposite image 300. In FIG. 3, representation 304 is illustrated toinclude diagonal lines, and representation 306 is illustrated to includevertical lines to indicate that representations 304 and 306 representthe surgical space as captured by different imaging modalities.

In composite image 300, representation 304 of the first portion of thesurgical space as captured by the first imaging modality is augmentedwith integrated representation 306 of the second portion of the surgicalspace as captured by the second imaging modality. Accordingly, a surgeonand/or other surgical team member viewing composite image 300 mayconcurrently visualize integrated representations 304 and 306 ofportions of the surgical space as captured by different imagingmodalities. Because representation 306 is positionally integrated withinrepresentation 304, the surgeon may visualize the surgical space ascaptured by the different imaging modalities without having to mentallyalign the representations 304 and 306 to one another and the surgicalspace as would be required if representations 304 and 306 were presentedseparately and were not positionally integrated with one another.

In certain examples, augmentation region 302 may be movable relative tothe surgical space by way of user input to a computer-assisted surgicalsystem. The computer-assisted surgical system may be configured toreceive any suitable user input that may be used to move augmentationregion 302 relative to the surgical space. Such input may includeactuation of buttons, movement of a controller (e.g., a joystickcontroller, a master control, etc.), movement of a surgical instrumentconnected to the computer-assisted surgical system (e.g., movement of anultrasound probe or other surgical instrument from which augmentationregion 302 is projected), and/or any other suitable user input.

Such movement of augmentation region 302 may allow a user of thecomputer-assisted surgical system to select, on the fly during asurgical procedure, a particular portion of the surgical space that isto be viewed as captured by the second imaging modality instead of ascaptured by the first imaging modality. This may allow a surgeon todynamically “spotlight” a select portion of the surgical space in orderto view the select portion as captured by the second imaging modality.For example, representation 304 may represent surface anatomy of apatient as captured by an endoscope, and representation 306 mayrepresent subsurface anatomy of the patient as captured by a differentimaging modality such as an ultrasound, CT, or MRI device. In thisexample, the surgeon may position augmentation region 302 to viewsubsurface anatomy at a select portion of the surgical space, whilestill viewing surface anatomy at another portion of the surgical space.In such implementations, augmentation region 302 may function as avirtual cut-away region (e.g., a virtual cut plane) that may be used bya surgeon to select a portion of a representation of surface anatomy tobe virtually cut away from view to reveal a representation of subsurfaceanatomy located behind the surface anatomy (e.g., a virtual cut planeinto a preoperative 3D model of the surgical space that is registeredwith an endoscopic view of the surgical space).

Movement of augmentation region 302 relative to the surgical space mayinclude movement in any suitable direction(s) relative to the surgicalspace. For example, the movement may include lateral movement that pansaugmentation region 302 across an image of the surgical space.Additionally or alternatively, the movement may include depth movementthat changes a distance of augmentation region 302 from the perspectiveviewpoint from which the image of the surgical space is rendered. Suchdepth movement of augmentation region 302 may position augmentationregion 302 at different depths relative to the surgical space, whichposition may be used to identify a slice of a virtual representation ofthe surgical space to be rendered (e.g., a slice of a 3D model that ismapped to a virtual representation of the surgical space). Such freedomof movement of augmentation region 302 may provide a user of thecomputer-assisted surgical system flexibility to select, on the flyduring a surgical procedure, a particular portion of the surgical spaceto be augmented and imagery to be used for the augmentation.

To generate a composite image of a surgical space, system 100 maydetermine an image render viewpoint from which to render an image of thesurgical space and, from a perspective of the image render viewpoint, aposition of an augmentation region relative to the surgical space.System 100 may generate a composite image of the surgical space from theperspective of the image render viewpoint and based on the determinedposition of the augmentation region relative to the surgical space. Thecomposite image may include: the augmentation region at the determinedposition of the augmentation region relative to the surgical space;outside the augmentation region, a representation of a first portion ofthe surgical space as captured by a first imaging modality; and insidethe augmentation region, a representation of a second portion of thesurgical space as captured by a second imaging modality. Therepresentation of the second portion of the surgical space may begenerated in any of the ways described herein and may be based on firstimagery of the surgical space captured by the first imaging modality andsecond imagery of the surgical space captured by the second imagingmodality.

FIG. 4 illustrates an exemplary depiction of a surgical space 400 withan image render viewpoint 402 and augmentation region 404 positionedrelative to surgical space 400. While the depiction shows atwo-dimensional (2D) view, principles described with respect to the 2Dview also apply to a 3D view of a surgical space with an image renderviewpoint 402 and an augmentation region 404 positioned relative to thesurgical space.

As shown in FIG. 4, surgical space 400 includes an anatomical structure406, which includes surface anatomy 408 and subsurface anatomy 410.Subsurface anatomy 410 may include any anatomy positioned behind surfaceanatomy 408 from the perspective of image render viewpoint 402 and/orhidden from view from the perspective of image render viewpoint 402 bysurface anatomy 408. In certain examples, surface anatomy 408 mayinclude an outer layer of tissue of a patient, and subsurface anatomy410 may include anatomy embedded within the outer layer of tissue.

Image render viewpoint 402 (“viewpoint 402”) may be any viewpoint fromwhich an image of surgical space 400 may be rendered. Viewpoint 402 mayinclude an actual viewpoint of an imaging modality such as an endoscope(e.g., a viewpoint of one or more cameras of an endoscope).Alternatively, viewpoint 402 may include a virtual viewpointcorresponding to an actual viewpoint of an imaging modality such as anendoscope. Viewpoint 402 may be associated with and/or representintrinsic and extrinsic properties of an imaging device such as one ormore cameras of an endoscope. Viewpoint 402 may have a field of viewwithin which an image of surgical space 400 may be rendered. A spacewithin solid-line arrows 412 extending from viewpoint 400 represents afield of view of viewpoint 402.

As shown, viewpoint 402 is located at a position relative to surgicalspace 400 and defines a perspective from which an image of surgicalspace 400 may be rendered. The position of viewpoint 402 illustrated inFIG. 4 is exemplary only. Viewpoint 402 may be located at any positionrelative to surgical space 400 from which an image of surgical space 400may be rendered.

Augmentation region 404 may be positioned relative to surgical space 400to define a portion of surgical space 400 that is to be augmented.Augmentation region 404 may be defined to include any suitable shape,area, or volume that may be positioned relative to surgical space 400 todelineate, from the perspective of viewpoint 402, a portion of surgicalspace 400 that is to be augmented. In FIG. 4, augmentation region 404 isrepresented as a side view of a planar shape, which may be a circle,oval, quadrilateral, or any other suitable planar shape.

In certain examples, augmentation region 404 may be a virtual object ora view of a virtual object from the perspective of viewpoint 402. Forexample, a virtual object may be defined and positioned relative tosurgical space. The virtual object may be a 2D or 3D object. The virtualobject may be movable relative to surgical space 400 by way of userinput to a computer-assisted surgical system.

At a given point in time, system 100 may determine positions ofviewpoint 402 and augmentation region 404 relative to surgical space 400and generated a composite image of surgical space 400 based on thedetermined positions of viewpoint 402 and augmentation region 404relative to surgical space 400. In the composite image, a first portionof surgical space 400 may be represented with imagery as captured by afirst imaging modality, and a second portion of surgical space 400 maybe represented with imagery as captured by a second imaging modality.

To this end, system 100 may use the determined positions of viewpoint402 and augmentation region 404 relative to surgical space 400 to definethe first and second portions of surgical space 400. To illustrate, FIG.4 shows dashed lines extending from viewpoint 402 and intersectingboundaries of augmentation region 404 to define alignment boundaries414.

Portions of surgical space 400 that are outside of a space withinalignment boundaries 414 are referred to as unaligned portions 416 ofsurgical space 400 because these portions are not aligned withaugmentation region 404 from the perspective of viewpoint 402. Unalignedportions 416 of surgical space 400 may make up a first portion ofsurgical space 400 in a composite image of surgical space 400.

A portion of surgical space 400 that is inside a space within alignmentboundaries 414 is referred to as an aligned portion 418 of surgicalspace 400 because this portion is aligned with augmentation region 404from the perspective of viewpoint 402. Aligned portion 418 of surgicalspace 400 may make up a second portion of surgical space 400 in thecomposite image of surgical space 400.

In the composite image of surgical space 400, a representation of thefirst portion of surgical space 400 (a representation of unalignedportions 416 of surgical space 400) may be generated based on imagerycaptured by a first imaging modality. For example, the first imagingmodality may include an endoscope positioned at viewpoint 402 to captureendoscopic imagery of surgical space 400, and the representation of thefirst portion of surgical space 400 may include endoscopic imagery ofsurface anatomy 408 in the unaligned portions 416 of surgical space 400.

In the composite image of surgical space 400, a representation of thesecond portion of surgical space 400 (a representation of alignedportion 418 of surgical space 400) may be generated based on imagerycaptured by a second imaging modality that is different from the firstimaging modality. For example, the second imaging modality may includean ultrasound, CT, or MRI device that captured ultrasound, CT, or MRIimagery of surgical space 400, and the representation of the secondportion of surgical space 400 may include ultrasound, CT, or MRI imageryof surface anatomy 408 or subsurface anatomy 410 in the aligned portion418 of surgical space 400. In examples in which the representation ofthe second portion of surgical space 400 is generated based on imageryof subsurface anatomy 410 in the aligned portion 418 of surgical space400, the imagery may be of subsurface anatomy 410 and any depth ordepths behind the surface anatomy 408. In some implementations, a depthof imagery of the subsurface anatomy 410 as captured by the secondimaging modality may be selected (e.g., on the fly during a surgicalprocedure) by user input to a computer-assisted surgical system, such asby user input that moves augmentation region 404 in a manner thatchanges the distance between viewpoint 402 and augmentation region 404and/or moves augmentation region 404 to a select depth within subsurfaceanatomy 410.

As described herein, the representation of the second portion ofsurgical space 400 in the composite image may be generated based onimagery captured by the second imaging modality and imagery captured bythe first imaging modality. For example, the representation of thesecond portion of surgical space 400 may include a composition ofimagery of subsurface anatomy 410 within aligned region 418 as capturedby the second imaging modality and a feature of imagery of surfaceanatomy 408 within aligned region 418 as captured by the first imagingmodality. For instance, system 100 may extract gradient information fromthe imagery of surface anatomy 408 within aligned region 418 andgenerate slope imagery representing the extracted gradient information.System 100 may modify the imagery of subsurface anatomy 410 withinaligned region 418 as captured by the second imaging modality with theslope imagery, such as by summing the extracted slope imagery and theimagery of subsurface anatomy 410 within aligned region 418 as capturedby the second imaging modality to generate a composition for therepresentation of the second portion of surgical space 400 in thecomposite image. As described, the composition may provide a visuallyrealistic representation of depth of subsurface anatomy 410 relative tosurface anatomy 408 in the composite image.

In certain examples, the combining of the slope imagery extracted fromimagery of surface anatomy 408 with the imagery of subsurface anatomy410 may be performed using actual, organic color values of the imageryof surface anatomy 408 and/or subsurface anatomy 410, without usingartificial or non-photorealistic colors. This may contribute to thevisually realistic representation of the surgical space, including thevisually realistic representation of depth of subsurface anatomy 410relative to surface anatomy 408 in the composite image.

An exemplary way of generating a composite image of a surgical spacewill now be described. FIG. 5A illustrates a real workspace 502 that maybe included in a surgical space. The real workspace 502 may be areal-world physical workspace located in front of an endoscope 504configured to capture endoscopic imagery of the workspace. In certainexamples, the real workspace 502 may include anatomy 506 of a patientand one or more surgical instruments 508 (e.g., surgical instruments508-1 and 508-2) positioned relative to anatomy 506 in the realworkspace 502. In the illustrated example, surgical instrument 508-1 isa grasper tool, and surgical instrument 508-2 is an ultrasound probe.

System 100 may access a real image of the real workspace 502. Forexample, system 100 may access a real image of the real workspace 502 ascaptured by endoscope 504. FIG. 5B illustrates an example of a realimage (R) of the real workspace 502 as captured by endoscope 504.

System 100 may generate a slope image from real image (R). For example,system 100 may extract gradient information from real image (R) and usethe extracted gradient information to generate a slope image thatrepresents the gradient information extracted from real image (R). Thegradient information may represent directional change in a feature ofreal image (R), such as a directional change in intensity, color, oranother feature of real image (R). The gradient information mayrepresent change in one or more directions.

FIG. 5C illustrates an example of a slope image (S) extracted from realimage (R). Slope image (S) is illustrated in black and white in FIG. 5C.However, slope image (S) may include black, white, and/or gray invarious shades that represent degrees of slope. Slope image (S) mayrepresent any suitable gradient information, including horizontalgradient, vertical gradient, one or more other directional gradients, orany combination or sub-combination thereof.

System 100 may generate and maintain a virtual workspace representativeof the real workspace 502. The virtual workspace may be a 3D space(e.g., a 3D coordinate space) to which imagery of the real workspace 502as captured by different imaging modalities may be mapped. For example,endoscopic imagery captured by endoscope 504 and other imagery capturedby one or more other imaging modalities may be mapped to the virtualworkspace such that the endoscopic imagery and the other imagery areregistered to one another in the virtual workspace.

The registration may be performed in any suitable way. For example,depth values may be determined and associated with pixels in real image(R) to generate a 3D mesh of 3D coordinate points that are associatedwith color values of the pixels. The 3D mesh may be mapped to thevirtual workspace.

In examples in which other imagery captured by another imaging modalityincludes a 3D model of a surgical space, system 100 may map the 3D modelto the virtual workspace. This may be performed in any suitable way andmay include system 100 registering features in the 3D model to matchingfeatures of the 3D mesh generated from endoscopic imagery and depthinformation associated with the endoscopic imagery. Accordingly, the 3Dmodel may be registered to the 3D mesh in the virtual workspace.

In examples in which other imagery captured by another imaging modalityincludes a 2D image of a surgical space and depth data for the 2D imageis available, system 100 may map the 2D image to the virtual workspacesimilarly to how system 100 maps endoscopic imagery to the virtualworkspace. In examples in which other imagery captured by anotherimaging modality includes a 2D image of a surgical space but depth datafor the 2D image is unavailable, system 100 may project the 2D image toany suitable surface in the virtual workspace, such as the surface of avirtual object representing an augmentation region in the virtualworkspace.

FIG. 6A illustrates an exemplary virtual workspace 602 that may begenerated by system 100 based on and/or representative of the realworkspace 502. The virtual workspace 602 may include virtualrepresentations of elements of the real workspace 502, such as a virtualrepresentation 606 of anatomy 506 and virtual representations 608 (e.g.,virtual representations 608-1 and 608-2) of surgical instruments 508(e.g., surgical instruments 508-1 and 508-2). System 100 may generatethe virtual representations in the virtual workspace in any suitableway, including based on real image (R) of the real workspace 502 anddepth information associated with real image (R).

The virtual workspace 602 may also include an image render viewpoint 604(“viewpoint 604”), which may be a virtual viewpoint corresponding to theviewpoint of endoscope 504 included in the real workspace 502. Viewpoint604 may be configured based on intrinsic and extrinsic properties ofendoscope 504. Viewpoint 604 may be positioned relative to otherelements of the virtual workspace 602 and may represent a viewpointperspective from which an image of the virtual workspace 602 may berendered.

The virtual workspace 602 may also include an augmentation region 610positioned relative to other elements of the virtual workspace 602. Inthe example illustrated in FIG. 6A, the position of augmentation region610 is determined based on ultrasound probe 608-2 by projectingaugmentation region 610 from ultrasound probe 608-2 in a particularmanner (e.g., in a particular direction, orientation, pose, shape, etc.)such that augmentation region 610 is positioned relative to ultrasoundprobe 608-2, viewpoint 604, and other elements of the virtual workspace602.

System 100 may project imagery captured by ultrasound probe 608-2 ontoaugmentation region 610 in the virtual workspace 602. The ultrasoundimagery projected onto augmentation region 610 is represented byhorizontal-line fill pattern 612 in FIG. 6A. While the exampleillustrated in FIG. 6A depicts a projection of ultrasound imagery ontoaugmentation region 610, any other imagery captured by another imagingmodality may be projected onto augmentation region 610 in the virtualworkspace 602. For example, system 100 may determine a slice of a 3Dmodel (e.g., a 3D model generated from CT or MRI imagery) based on aposition of augmentation region 610 relative to the 3D model registeredto the virtual workspace 602 and project an image of the slice ontoaugmentation region 610. In such an example, augmentation region 610projected from ultrasound probe 608-2 may function as a placeholder ontowhich an image of a slice of a registered 3D model of the surgical space602 may be projected in the virtual workspace 602.

In certain examples, an image that is projected onto augmentation region610 in the virtual workspace may be selected by way of user input to acomputer-assisted surgical system. For example, a user of thecomputer-assisted surgical system may provide input to toggle from oneimaging modality image being projected onto augmentation region 610 toanother imaging modality image being projected onto augmentation region610 (e.g., from an ultrasound image to a CT or MRI model image or viceversa).

System 100 may be configured to use the virtual workspace 602 in anysuitable way to generate a composite image of the surgical space. Forexample, system 100 may generate an image of the virtual workspace 602from the perspective of viewpoint 604. FIG. 6B illustrates an example ofan image (V), which may be referred to as a virtual image (V), of thevirtual workspace 602 rendered by system 100 from the perspective ofviewpoint 604. As described herein, system 100 may be configured to usevirtual image (V) to generate a composite image of the surgical space.

System 100 may generate a mask image such as a binary render mask imagebased on the virtual workspace 602 and/or virtual image (V). The maskimage may correspond in size to virtual image (V) and may include afirst portion that is aligned with the position of augmentation region610 (e.g., an area inside of augmentation region 610) and that isassigned a first binary value, and a second portion that is not alignedwith the position of augmentation region 610 (e.g., an area outside ofaugmentation region 610) and that is assigned a second binary valuedifferent from the first binary value. FIG. 6C illustrates an example ofa mask image (M) that may be generated by system 100 based on virtualimage (V). As shown, mask image (M) includes a first portion 622including a white fill representing one binary value, and a secondportion 624 including a black fill representing another binary value.

System 100 may be configured to use real image (R), slope image (S),virtual image (V), and mask image (M) to generate a composite image (C)of the surgical space. For example, system 100 may perform a blendfunction that generates a composite image (C) based on the real image(R), slope image (S), virtual image (V), and mask image (M). In certainimplementations, the following blend function may be performed by system100 for i, j iterating over the image width and height, respectively:

C=blend(R,S,V,M)=M _((i,j))*(S _((i,j)) +V _((i,j)))+(1−M_((i,j)))*R_((i,j))

In this blend function, mask image (M) may have a binary value of “1” ina first portion that is aligned with augmentation region 610 and abinary value of “0” in a second portion that is not aligned withaugmentation region 610. Accordingly, system 100 may use real image (R)for all pixel locations that are set to “0” in mask image (M) and mayuse a blended output of virtual image (V) and slope image (S) for allpixel locations that are set to “1” in mask image (M).

Based on this blend function, the first portion of the composite image(C) may include a representation of the surgical scene as captured byendoscope 504, and the second portion of the composite image (C) mayinclude a representation of the surgical scene as captured by anotherimaging modality. The representation of the second portion of thesurgical scene as captured by another imaging modality may include acomposition of imagery of the surgical space as captured by the otherimaging modality and a feature of the endoscopic imagery of the surgicalspace. For instance, to produce the representation of the second portionof the surgical space, system 100 may combine the imagery of the secondportion of the surgical space as captured by the other imaging modality(e.g., this may include the imagery as mapped to the augmentation region610 in the virtual workspace 602 and/or as represented in virtual image(V)) and slope imagery representing gradient information extracted fromthe endoscopic imagery of the second portion of the surgical space. Thiscombination may be performed by system 100 executing the blend functioniterating over the pixel locations of the composite image (C) andselectively using values at corresponding pixel locations in real image(R), slope image (S), virtual image (V), and mask image (M), asindicated by the blend function, to generate the composite image (C). Incertain examples, this may be performed without using artificial colorsto generate composite image (C).

FIG. 7 illustrates an example of a composite image (C) that may begenerated by system 100 performing a blend function to generatecomposite image (C) based on the real image (R), slope image (S),virtual image (V), and mask image (M). As shown, composite image (C) mayinclude a representation 702 of augmentation region 610. Outside therepresentation 702 of augmentation region 610, composite image (C)includes a representation 704 of a first portion of the surgical space,which representation includes endoscopic imagery of surface anatomy ascaptured by endoscope 504. Inside the representation 702 of augmentationregion 610, composite image (C) includes a representation 706 of asecond portion of the surgical space as captured by another imagingmodality (e.g., an ultrasound probe), which representation includes acomposition of imagery captured by the other imaging modality and afeature of the endoscopic imagery (e.g., a slope of the endoscopicimagery) of the second region of the surgical space.

A composite image, such as composite image (C), may include any suitablerepresentation of an augmentation region. Such a representation mayinclude any visual representation of a boundary or transition betweenrepresentations of first and second portions of a surgical scene in thecomposite image.

While FIG. 5A-FIG. 7 illustrate an example in which a first imagingmodality includes an endoscope and a second imaging modality includes anultrasound probe, images as captured by other different imagingmodalities may be similarly processed to generate a composite image of asurgical space. In other examples, for instance, a first imagingmodality may include an endoscope, and a second imaging modality mayinclude a CT or MRI machine. In such examples, images may be captured bythe CT or MRI machine during a preoperative phase of a surgicalprocedure and processed to generate a 3D model of the surgical space.System 100 may register the 3D model of the surgical space to theendoscopic imagery of the surgical space in a virtual workspace. System100 may determine a position of an image render viewpoint relative tothe surgical space, determine a position of an augmentation regionrelative to the surgical space, and use the position of the image renderviewpoint and the position of the augmentation region relative to thesurgical space to identify an image of the 3D model (e.g., a planarslice of the 3D image) to be used to generate a representation of thesurgical space within the augmentation region. For example, system 100may project the identified image of the 3D model onto the augmentationregion in the virtual workspace and perform a blend function asdescribed herein to generate a composite image of the surgical space.

In certain implementations, for example, system 100 may project anaugmentation region from an ultrasound probe positioned in the surgicalspace. To this end, system 100 may access tracking information (e.g.,position information, orientation information, movement information,kinematic information, etc.) for the ultrasound probe from acomputer-assisted surgical system to which the ultrasound probe isconnected and use the tracking information to identify a pose of theultrasound probe (e.g., a position and an orientation of the ultrasoundprobe) within the surgical space. System 100 may project theaugmentation region into the surgical space (e.g., into a virtualworkspace representing a real workspace within the surgical space) basedon the pose of the ultrasound probe.

In certain examples, system 100 may generate a composite image of thesurgical space that includes, within the augmentation region, arepresentation of ultrasound imagery of the surgical space as capturedby the ultrasound probe. In other examples, system 100 may generate acomposite image of the surgical space that includes, within theaugmentation region, a representation of a portion of the surgical spaceas captured by a different imaging modality. For example, therepresentation within the augmentation region may include or may bebased on CT or MRI imagery of the surgical space (e.g., a 3D model ofthe surgical space generated from CT or MRI imaging of the surgicalspace). In such an example, the augmentation region projected from theultrasound probe may function as a virtual cut-away region and/or aplaceholder on which to project the CT-based or MRI-based representationof the surgical space. Accordingly, a user of a computer-assistedsurgical system may provide input to position the ultrasound probewithin the surgical space to select a position of the augmentationregion to define a portion of the surgical space to be augmented with animage of a registered 3D model of the surgical space. As mentioned, incertain examples, system 100 may be configured to toggle therepresentation within the augmentation region between representing theultrasound imagery as captured by the ultrasound probe and representingother imagery as captured by another imaging modality (e.g., CT or MRIimagery captured by a CT or MRI machine).

Use of an ultrasound probe to define and move an augmentation regionrelative to the surgical space is illustrative of certain examples.Other examples may implement a different real-world surgical instrument,a virtual object, or any other suitable mechanism to be used by a userto define and move an augmentation region relative to the surgicalspace.

As mentioned, system 100 may be implemented in or communicativelycoupled to a computer-assisted surgical system. System 100 may receiveinput from and provide output to the computer-assisted surgical system.For example, system 100 may access imagery of a surgical space and/orany information about the surgical space and/or the computer-assistedsurgical system from the computer-assisted surgical system, use theaccessed imagery and/or information to perform any of the processingdescribed herein to generate composite imagery of the surgical space,and provide data representative of the composite imagery to thecomputer-assisted surgical system for display (e.g., by a display deviceassociated with the computer-assisted surgical system).

FIG. 8 illustrates an exemplary computer-assisted surgical system 800(“surgical system 800”). System 100 may be implemented by surgicalsystem 800, connected to surgical system 800, and/or otherwise used inconjunction with surgical system 800.

As shown, surgical system 800 may include a manipulating system 802, auser control system 804, and an auxiliary system 806 communicativelycoupled one to another. Surgical system 800 may be utilized by asurgical team to perform a computer-assisted surgical procedure on apatient 808. As shown, the surgical team may include a surgeon 810-1, anassistant 810-2, a nurse 810-3, and an anesthesiologist 810-4, all ofwhom may be collectively referred to as “surgical team members 810.”Additional or alternative surgical team members may be present during asurgical session as may serve a particular implementation.

While FIG. 8 illustrates an ongoing minimally invasive surgicalprocedure, it will be understood that surgical system 800 may similarlybe used to perform open surgical procedures or other types of surgicalprocedures that may similarly benefit from the accuracy and convenienceof surgical system 800. Additionally, it will be understood that thesurgical session throughout which surgical system 800 may be employedmay not only include an operative phase of a surgical procedure, as isillustrated in FIG. 8, but may also include preoperative, postoperative,and/or other suitable phases of the surgical procedure.

As shown in FIG. 8, manipulating system 802 may include a plurality ofmanipulator arms 812 (e.g., manipulator arms 812-1 through 812-4) towhich a plurality of surgical instruments may be coupled. Each surgicalinstrument may be implemented by any suitable surgical tool (e.g., atool having tissue-interaction functions), medical tool, imaging device(e.g., an endoscope, an ultrasound tool, etc.), sensing instrument(e.g., a force-sensing surgical instrument), diagnostic instrument, orthe like that may be used for a computer-assisted surgical procedure onpatient 808 (e.g., by being at least partially inserted into patient 808and manipulated to perform a computer-assisted surgical procedure onpatient 808). While manipulating system 802 is depicted and describedherein as including four manipulator arms 812, it will be recognizedthat manipulating system 802 may include only a single manipulator arm812 or any other number of manipulator arms as may serve a particularimplementation.

Manipulator arms 812 and/or surgical instruments attached to manipulatorarms 812 may include one or more displacement transducers, orientationalsensors, and/or positional sensors used to generate raw (i.e.,uncorrected) kinematics information. One or more components of surgicalsystem 800 may be configured to use the kinematics information to track(e.g., determine positions of) and/or control the surgical instruments.

User control system 804 may be configured to facilitate control bysurgeon 810-1 of manipulator arms 812 and surgical instruments attachedto manipulator arms 812. For example, surgeon 810-1 may interact withuser control system 804 to remotely move or manipulate manipulator arms812 and the surgical instruments. To this end, user control system 804may provide surgeon 810-1 with imagery (e.g., high-definition 3Dimagery) of a surgical space associated with patient 808 as captured byan imaging system (e.g., any of the medical imaging systems describedherein). In certain examples, user control system 804 may include astereo viewer having two displays where stereoscopic images of asurgical space associated with patient 808 and generated by astereoscopic imaging system may be viewed by surgeon 810-1. In certainexamples, composite imagery generated by system 100 may be displayed byuser control system 804. Surgeon 810-1 may utilize the imagery displayedby user control system 804 to perform one or more procedures with one ormore surgical instruments attached to manipulator arms 812.

To facilitate control of surgical instruments, user control system 804may include a set of master controls. These master controls may bemanipulated by surgeon 810-1 to control movement of surgical instruments(e.g., by utilizing robotic and/or teleoperation technology). The mastercontrols may be configured to detect a wide variety of hand, wrist, andfinger movements by surgeon 810-1. In this manner, surgeon 810-1 mayintuitively perform a procedure using one or more surgical instruments.

Auxiliary system 806 may include one or more computing devicesconfigured to perform primary processing operations of surgical system800. In such configurations, the one or more computing devices includedin auxiliary system 806 may control and/or coordinate operationsperformed by various other components (e.g., manipulating system 802 anduser control system 804) of surgical system 800. For example, acomputing device included in user control system 804 may transmitinstructions to manipulating system 802 by way of the one or morecomputing devices included in auxiliary system 806. As another example,auxiliary system 806 may receive, from manipulating system 802, andprocess image data representative of imagery captured by an imagingdevice attached to one of manipulator arms 812.

In some examples, auxiliary system 806 may be configured to presentvisual content to surgical team members 810 who may not have access tothe images provided to surgeon 810-1 at user control system 804. To thisend, auxiliary system 806 may include a display monitor 814 configuredto display one or more user interfaces, such as images (e.g., 2D images)of the surgical space, information associated with patient 808 and/orthe surgical procedure, and/or any other visual content as may serve aparticular implementation. For example, display monitor 814 may displayimages of the surgical space (e.g., composite images generated by system100) together with additional content (e.g., graphical content,contextual information, etc.) concurrently displayed with the images. Insome embodiments, display monitor 814 is implemented by a touchscreendisplay with which surgical team members 810 may interact (e.g., by wayof touch gestures) to provide user input to surgical system 800.

Manipulating system 802, user control system 804, and auxiliary system806 may be communicatively coupled one to another in any suitablemanner. For example, as shown in FIG. 8, manipulating system 802, usercontrol system 804, and auxiliary system 806 may be communicativelycoupled by way of control lines 816, which may represent any wired orwireless communication link as may serve a particular implementation. Tothis end, manipulating system 802, user control system 804, andauxiliary system 806 may each include one or more wired or wirelesscommunication interfaces, such as one or more local area networkinterfaces, Wi-Fi network interfaces, cellular interfaces, etc.

FIG. 9 shows an exemplary method 900. While FIG. 9 illustrates exemplaryoperations according to one embodiment, other embodiments may omit, addto, reorder, combine, and/or modify any of the steps shown in FIG. 9 Oneor more of the operations shown in in FIG. 9 may be performed by acomputing system such as system 100, any components included therein,and/or any implementation thereof.

In operation 902, a computing system determines an image renderviewpoint from which to render an image of a surgical space. Operation902 may be performed in any of the ways described herein.

In operation 904, the computing system determines, from a perspective ofthe image render viewpoint, a position of an augmentation regionrelative to the surgical space. Operation 904 may be performed in any ofthe ways described herein.

In operation 906, the computing system generates a composite image ofthe surgical space from the perspective of the image render viewpointand based on the determined position of the augmentation region relativeto the surgical space. Operation 906 may be performed in any of the waysdescribed herein.

In operation 908, the computing system directs a display device todisplay the composite image. Operation 908 may be performed in any ofthe ways described herein.

The composite image generated in operation 806 may be generated in anysuitable way, including in any of the ways described herein. Thecomposite image may include any of the exemplary elements describedherein. For example, the composite image may include the augmentationregion at the determined position of the augmentation region relative tothe surgical space. Outside the augmentation region, the composite imagemay include a representation of a first portion of the surgical space ascaptured by a first imaging modality. Inside the augmentation region,the composite image may include a representation of a second portion ofthe surgical space as captured by a second imaging modality.

As described herein, the representation of a first portion of thesurgical space as captured by a first imaging modality may includeand/or may be generated based on the first imagery of the surgical spacecaptured by the first imaging modality (e.g., an endoscope), and therepresentation of the second portion of the surgical space may begenerated based on the first imagery of the surgical space captured bythe first imaging modality and the second imagery of the surgical spacecaptured by the second imaging modality (e.g., an ultrasound probe, a CTdevice, or an MRI device). For example, as described herein, therepresentation of the second portion of the surgical space may include acomposition of the second imagery of the surgical space captured by thesecond imaging modality and a feature of the first imagery of thesurgical space captured by the first imaging modality.

In some examples, a non-transitory computer-readable medium storingcomputer-readable instructions may be provided in accordance with theprinciples described herein. The instructions, when executed by aprocessor of a computing device, may direct the processor and/orcomputing device to perform one or more operations, including one ormore of the operations described herein. Such instructions may be storedand/or transmitted using any of a variety of known computer-readablemedia.

A non-transitory computer-readable medium as referred to herein mayinclude any non-transitory storage medium that participates in providingdata (e.g., instructions) that may be read and/or executed by acomputing device (e.g., by a processor of a computing device). Forexample, a non-transitory computer-readable medium may include, but isnot limited to, any combination of non-volatile storage media and/orvolatile storage media. Exemplary non-volatile storage media include,but are not limited to, read-only memory, flash memory, a solid-statedrive, a magnetic storage device (e.g. a hard disk, a floppy disk,magnetic tape, etc.), ferroelectric random-access memory (“RAM”), and anoptical disc (e.g., a compact disc, a digital video disc, a Blu-raydisc, etc.). Exemplary volatile storage media include, but are notlimited to, RAM (e.g., dynamic RAM).

FIG. 10 illustrates an exemplary computing device 1000 that may bespecifically configured to perform one or more of the processesdescribed herein. Any of the systems, units, computing devices, and/orother components described herein may be implemented by computing device1000.

As shown in FIG. 10, computing device 1000 may include a communicationinterface 1002, a processor 1004, a storage device 1006, and aninput/output (“I/O”) module 1008 communicatively connected one toanother via a communication infrastructure 1010. While an exemplarycomputing device 1000 is shown in FIG. 10, the components illustrated inFIG. 10 are not intended to be limiting. Additional or alternativecomponents may be used in other embodiments. Components of computingdevice 1000 shown in FIG. 10 will now be described in additional detail.

Communication interface 1002 may be configured to communicate with oneor more computing devices. Examples of communication interface 1002include, without limitation, a wired network interface (such as anetwork interface card), a wireless network interface (such as awireless network interface card), a modem, an audio/video connection,and any other suitable interface.

Processor 1004 generally represents any type or form of processing unitcapable of processing data and/or interpreting, executing, and/ordirecting execution of one or more of the instructions, processes,and/or operations described herein. Processor 1004 may performoperations by executing computer-executable instructions 1012 (e.g., anapplication, software, code, and/or other executable data instance)stored in storage device 1006.

Storage device 1006 may include one or more data storage media, devices,or configurations and may employ any type, form, and combination of datastorage media and/or device. For example, storage device 1006 mayinclude, but is not limited to, any combination of the non-volatilemedia and/or volatile media described herein. Electronic data, includingdata described herein, may be temporarily and/or permanently stored instorage device 1006. For example, data representative ofcomputer-executable instructions 1012 configured to direct processor1004 to perform any of the operations described herein may be storedwithin storage device 1006. In some examples, data may be arranged inone or more databases residing within storage device 1006.

I/O module 1008 may include one or more I/O modules configured toreceive user input and provide user output. I/O module 1008 may includeany hardware, firmware, software, or combination thereof supportive ofinput and output capabilities. For example, I/O module 1008 may includehardware and/or software for capturing user input, including, but notlimited to, a keyboard or keypad, a touchscreen component (e.g.,touchscreen display), a receiver (e.g., an RF or infrared receiver),motion sensors, and/or one or more input buttons.

I/O module 1008 may include one or more devices for presenting output toa user, including, but not limited to, a graphics engine, a display(e.g., a display screen), one or more output drivers (e.g., displaydrivers), one or more audio speakers, and one or more audio drivers. Incertain embodiments, I/O module 1008 is configured to provide graphicaldata to a display for presentation to a user. The graphical data may berepresentative of one or more graphical user interfaces and/or any othergraphical content as may serve a particular implementation.

In the preceding description, various exemplary embodiments have beendescribed with reference to the accompanying drawings. It will, however,be evident that various modifications and changes may be made thereto,and additional embodiments may be implemented, without departing fromthe scope of the invention as set forth in the claims that follow. Forexample, certain features of one embodiment described herein may becombined with or substituted for features of another embodimentdescribed herein. The description and drawings are accordingly to beregarded in an illustrative rather than a restrictive sense.

1-20. (canceled)
 21. A system comprising: a memory storing instructions;and a processor communicatively coupled to the memory and configured toexecute the instructions to: generate a composite image of a surgicalspace from a perspective of an image render viewpoint and based on aposition of an augmentation region relative to the surgical space, thecomposite image comprising: the augmentation region at the position ofthe augmentation region relative to the surgical space, outside theaugmentation region, a representation of a first portion of the surgicalspace as captured by a first imaging modality, and inside theaugmentation region, a representation of a second portion of thesurgical space, the representation of the second portion generated basedon gradient information extracted from first imagery of the surgicalspace captured by the first imaging modality and on second imagery ofthe surgical space captured by the second imaging modality; and direct adisplay device to display the composite image.
 22. The system of claim21, wherein the representation of the second portion of the surgicalspace comprises a composition of the second imagery of the surgicalspace captured by the second imaging modality and the gradientinformation extracted from the first imagery of the surgical spacecaptured by the first imaging modality.
 23. The system of claim 22,wherein the gradient information represents directional change inintensity, color, or depth of the first imagery.
 24. The system of claim22, wherein the composition is generated by summing the second imageryof the surgical space captured by the second imaging modality and slopeimagery representing the gradient information extracted from the firstimagery of the surgical space.
 25. The system of claim 21, wherein thegradient information is extracted from a portion of the first imagerythat is within the augmentation region.
 26. The system of claim 21,wherein: the first imagery comprises imagery of surface anatomy includedin the first portion of the surgical space; and the second imagerycomprises imagery of subsurface anatomy included in the second portionof the surgical space.
 27. The system of claim 21, wherein: therepresentation of the first portion of the surgical space comprisesendoscopic imagery of surface anatomy included in the first portion ofthe surgical space; the gradient information extracted from the firstimagery comprises gradient imagery extracted from endoscopic imagery ofsurface anatomy included in the second portion of the surgical space;and the representation of the second portion of the surgical spacecomprises a composition of imagery of subsurface anatomy included in thesecond portion of the surgical space, and slope imagery representing thegradient information extracted from the endoscopic imagery of thesurface anatomy included in the second portion of the surgical space.28. The system of claim 21, wherein: the first imaging modalitycomprises an endoscopic imaging modality; and the second imagingmodality comprises one of an ultrasound imaging modality, a computerizedtomography (CT) imaging modality, or a magnetic resonance imaging (MRI)imaging modality.
 29. The system of claim 21, wherein: the first portionof the surgical space is unaligned with the augmentation region from theperspective of the image render viewpoint; and the second portion of thesurgical space is aligned with the augmentation region from theperspective of the image render viewpoint.
 30. The system of claim 21,wherein the augmentation region is projected from a surgical instrumentthat is selectively movable relative to the surgical space based on userinput to a computer-assisted surgical system.
 31. The system of claim21, wherein: the augmentation region is projected from an ultrasoundprobe that is selectively movable relative to the surgical space basedon user input to a computer-assisted surgical system; the first imagerycomprises endoscopic imagery of the surgical space; and the secondimagery comprises a slice image of a three-dimensional (3D) modelgenerated from preoperative computerized tomography (CT) imagery orpreoperative magnetic resonance imaging (MRI) imagery of the surgicalspace.
 32. The system of claim 21, wherein the generating of thecomposite image comprises: generating a virtual workspace representativeof a real workspace included in the surgical space, the virtualworkspace including the first imagery of the surgical space captured bythe first imaging modality and mapped to the virtual workspace, thesecond imagery of the surgical space captured by the second imagingmodality and mapped to the virtual workspace, a virtual viewpointrepresenting the image render viewpoint mapped to the virtual workspace,and a virtual object representing the augmentation region and positionedrelative to the first imagery, the second imagery, and the virtualviewpoint in the virtual workspace; and using the virtual workspace togenerate the composite image of the surgical space.
 33. The system ofclaim 32, wherein the using of the virtual workspace to generate thecomposite image of the surgical space comprises: generating a virtualimage of the virtual workspace; generating a slope image representingthe gradient information extracted from the first imagery of thesurgical space; and combining a portion of the virtual image of thevirtual workspace and a corresponding portion of the slope image togenerate the representation of the second portion of the surgical space.34. A system comprising: a memory storing instructions; and a processorcommunicatively coupled to the memory and configured to execute theinstructions to: generate a composite image of a surgical space based onfirst imagery captured by a first imaging modality and second imagerycaptured by the second imaging modality different from the first imagingmodality, the composite image comprising: a representation of a firstportion of the surgical space, the representation of the first portionof the surgical space generated based on the first imagery captured bythe first imaging modality, an augmentation region integrated within therepresentation of the first portion of the surgical space, and insidethe augmentation region, a representation of a second portion of thesurgical space, the representation of the second portion generated basedon a composition of the second imagery that is within the augmentationregion modified by gradient information extracted the first imagery thatis within the augmentation region; and direct a display device todisplay the composite image.
 35. The system of claim 34, wherein thecomposition is generated based on actual, organic colors included in thefirst imagery that is within the augmentation region and the secondimagery that is within the augmentation region, without using artificialcolors.
 36. A method comprising: generating, by a computing system, acomposite image of a surgical space from a perspective of an imagerender viewpoint and based on a position of an augmentation regionrelative to the surgical space, the composite image comprising: theaugmentation region at the position of the augmentation region relativeto the surgical space, outside the augmentation region, a representationof a first portion of the surgical space as captured by a first imagingmodality, and inside the augmentation region, a representation of asecond portion of the surgical space, the representation of the secondportion generated based on gradient information extracted from firstimagery of the surgical space captured by the first imaging modality andon second imagery of the surgical space captured by the second imagingmodality; and directing, by the computing system, a display device todisplay the composite image.
 37. The method of claim 36, wherein therepresentation of the second portion of the surgical space comprises acomposition of the second imagery of the surgical space captured by thesecond imaging modality and the gradient information extracted from thefirst imagery of the surgical space captured by the first imagingmodality.
 38. The method of claim 36, wherein: the representation of thefirst portion of the surgical space comprises endoscopic imagery ofsurface anatomy included in the first portion of the surgical space; thegradient information extracted from the first imagery comprises gradientimagery extracted from endoscopic imagery of surface anatomy included inthe second portion of the surgical space; and the representation of thesecond portion of the surgical space comprises a composition of imageryof subsurface anatomy included in the second portion of the surgicalspace, and slope imagery representing the gradient information extractedfrom the endoscopic imagery of the surface anatomy included in thesecond portion of the surgical space.
 39. The method of claim 36,wherein the augmentation region is projected from a surgical instrumentthat is selectively movable relative to the surgical space based on userinput to a computer-assisted surgical system.
 40. The method of claim36, wherein: the augmentation region is projected from an ultrasoundprobe that is selectively movable relative to the surgical space basedon user input to a computer-assisted surgical system; the first imagerycomprises endoscopic imagery of the surgical space; and the secondimagery comprises a slice image of a three-dimensional (3D) modelgenerated from preoperative computerized tomography (CT) imagery orpreoperative magnetic resonance imaging (MRI) imagery of the surgicalspace.